AUTOMORPHIC EQUIVALENCE PROBLEM FOR 
FREE ASSOCIATIVE ALGEBRAS OF RANK TWO 



VESSELIN DRENSKY AND JIE-TAI YU 

Abstract. Let K{x,y) be the free associative algebra of rank 2 
over an algebraically closed constructive field of any characteristic. 
We present an algorithm which decides whether or not two ele- 
ments in K(x, y) are equivalent under an automorphism of K{x, y). 
A modification of our algorithm solves the problem whether or not 
an element in K(x,y) is a semiinvariant of a nontrivial automor- 
phism. In particular, it determines whether or not the element has 
a nontrivial stabilizer in AutK(x, y). 

An algorithm for equivalence of polynomials under automor- 
phisms of C[x,y] was presented by Wightwick. Another, much 
simpler algorithm for automorphic equivalence of two polynomi- 
als in K[x,y) for any algebraically closed constructive field K was 
given by Makar-Limanov, Shpilrain, and Yu. In our approach we 
combine an idea of the latter three authors with an idea from the 
unpubished thesis of Lane used to describe automorphisms which 
stabilize elements of K(x,y). This also allows us to give a simple 
proof of the corresponding result for K[x,y] obtained by Makar- 
Limanov, Shpilrain, and Yu. 



1. Introduction 

Let K be an arbitrary field of any characteristic and let K[x, y] and 
K(x,y) be, respectively, the polynomial algebra in two variables and 
the free unitary associative algebra of rank 2 (or the algebra of poly- 
nomials in the noncommuting variables x and y). Two polynomials 
u(x,y) and v(x,y) from K[x, y] or K(x,y) are automorphically equiv- 
alent, if there exists an automorphism of the corresponding algebra 
which brings u to v. Wightwick has presented an algorithm which 
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decides whether or not two polynomials in C[x, y] are automorphically 
equivalent. Makar-Limanov, Shpilrain, and Yu ^Uj have given a much 
simpler algorithm which works for K[x,y] for any algebraically closed 
constructive K. Their method is based on peak reduction. See the sur- 
vey article of Shpilrain and Yu ^2] for other applications of the peak 
reduction method to problems for commutative algebra. Shpilrain and 
Yu [TT] have settled a special case of the automorphic equivalence prob- 
lem for K(x, y) , namely, the case where one of the elements is primitive. 

It is a classical result of Jung |Hj and van der Kulk [Zj that every au- 
tomorphism of K[x, y] is tame and is a product of two kind of automor- 
phisms - affine and triangular. Even more, AutK[x,y] is isomorphic 
to A*cB, the free product of the subgroup A of affine automorphisms 
and the subgroup B of triangular automorphisms amalgamating their 
intersection C, the subgroup of affine triangular automorphisms. This 
implies that every ip e AutK[x, y] has a canonical form ip = ip n ■ ■ ■ ipi, 
where each ipi is an affine or triangular automorphism, and the length 
n is invariant of ip. 

Let ip = ipx---ip n £ AutK[x,y] bring u(x,y) to v(x, y). Makar- 
Limanov, Shpilrain, and Yu (THj have studied the behaviour of the 
sequence 

di = max(deg a .(?/'i • • • ipiu), deg y (ipi ■ ■■ipiu)), i = 0, 1, . . . ,n, 

where deg x and deg y denote the degree with respect to x and y, respec- 
tively. If, at some step di < di + i > d i+2 (a peak), then they replace 
r/>i+i with another affine or triangular automorphism such that the 
new maximum d' i+1 of the degrees in x and y of ■ ■ ■ ip\U is smaller 

than di + \. In this way they move the peak to the right. This proce- 
dure gives that u(x, y) and v(x, y) are automorphically equivalent if and 
only if there exist two sequences of affine or triangular automorphisms, 
Pi,...,p r and o"i, . . . , a s , with the following property. The sequences 
of degrees p { = max(deg x (pi ■ • • pxu), deg y (pi ■ ■ ■ p±u)), i = 1, . . . , r, and 
qj = max(deg a .(crj • ■ ■ 0ii>), deg y (aj ■ ■ ■ o~iv)), j = 1, . . . , s, strictly de- 
crease, p r = q s , and there is an affine automorphism which sends 
p r ■ ■ ■ p\u to a s ■ ■ ■ o~\V. The procedure which decides whether or not 
such sequences of automorphisms exist reduces the problem to the deci- 
sion whether or not a system of algebraic equations in several variables 
is consistent. Over an algebraically closed constructive K this problem 
can be solved using Grobner bases techniques. 
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The K(x, y)-analogue of the theorem of Jung-van der Kulk has been 
established by Czerniakiewicz ^] and Makar-Limanov |9jj. Again, every 
automorphism is tame and AutK(x, y) is the free product with amal- 
gamation of the subgroups of triangular and affine automorphisms. 
Clearly, the automorphisms of K(x,y) fix, up to a nonzero multiplica- 
tive constant, the commutator [x, y] = xy — yx. 

A theorem of Lane from his unpublished thesis |H] in 1976 states 
that an automorphism <p of K(x, y) has a nontrivial semiinvariant (i.e., 
tpu = \u for some u(x, y) G K(x, y)\span([x, y] k \ k > 0) and a nonzero 
constant A G K) if and only if ip is conjugate in AutK(x, y) to a linear 
or triangular automorphism. See Section 9 of Chapter 6 from the book 
by Cohn [Hj for the improved exposition of the results of Lane. The 
idea of the proof is the following. Every cp — ip n ■ ■ • tpx G AutK(x, y) 
is written in a canonical form and the considerations are modulo the 
subspace spanned by the powers of the commutator [x,y]. The first 
step is to show that the consecutive action of nonaffine triangular au- 
tomorphisms ipi first strictly decrease the total degree of the element 
u(x,y). Then, maybe after one action, when the degree is the same, it 
starts to increase strictly. This allows to bound from above the length 
n in the canonical form of the automorphisms with u(x,y) as a semi- 
invariant. Then the proof is completed by arguments from the theory 
of free products of groups with amalgamation. 

Lane [Hj (see Exercise 6.9.3, p. 362 of has proved also that the 
only automorphisms of C[x, y] with semiinvariants u(x,y) G C[x,y]\C 
are conjugate to linear and triangular automorphisms. Smith jT3j has 
determined the eigenvalues and the eigenvectors of triangular automor- 
phisms. (Clearly, after a linear transformation of x and y, the linear 
automorphisms also become triangular.) Recently, the theorem of Lane 
has been generalized to any field K by Makar-Limanov, Shpilrain, and 
Yu [10J, involving algebraic geometry. 

In the present paper, by combining the algorithmic approach of 
Makar-Limanov, Shpilrain, and Yu jTUj to the automorphic equiva- 
lence in K[x,y] with the idea of Lane (as stated in [3]) in the de- 
scription of automorphisms of K(x, y) possessing nontrivial semiinvari- 
ants, we obtain an algorithm deciding whether or not two elements in 
K(x, y) are equivalent under an automorphism of K(x, y). A modifica- 
tion of our algorithm solves the problem whether or not an element in 
K(x,y) is a semiinvariant of a nontrivial automorphism. In particular, 
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it determines whether or not the element has a nontrivial stabilizer in 
AuiK(x, y). 

Our approach works also in the commutative case. We slightly im- 
prove the automorphic equivalence algorithm of Makar-Limanov, Sh- 
pilrain, and Yu ^0] , replacing the study of the behaviour of the degree 
with respect to x and y with that of the total degree. We simplify also 
the proof (over an arbitrary field K) of the result for the stabilizer of 
u(x,y) G K[x,y]\K, avoiding usage of algebraic geometry, and pro- 
vide an algorithm for the existence of a nontrivial stabilizer for a given 
u(x,y). 

2. Preliminaries 

Since automorphisms of K[x,y] and K{x,y) are determined by the 
images of x and y, we shall denote them as <p = (a,b), where <px = 
a(x, y), (py — b(x, y). If ip — (c, d) is another automorphism, we denote 
their composition as 

ijxp = (c, d)(a, b) = (a(c, d), b(c, d)). 

The automorphism ip is affine, if it is of the form 

?/) = (ax + 7?/ + f , (3x + 5y + rj), a, /3, 7, 5, £, rj E K. 

It is triangular, if 

V> = (ax + p(y),(3y + r)), a,(3eK* = K\0, rj G K, 

and the polynomial p(y) does not depend on x. We denote by A and 
B, respectively, the groups of affine and triangular automorphisms, and 
with C = A fl B their intersection. The results of Jung [Hj, van der 
Kulk Czerniakiewicz [4 , and Makar-Limanov || give that 

AxAK[x, y] = AutK(x, y) = A* c B. 
Hence (p G AutK[x, y] (and similarly for ip G AutK(x, y)) has the form 

(1) V 5 = Ipn- ■ ^1, 

where each ipi is affine or triangular. If two consequent ipi^i+i belong 
to the same A or B, we can replace them with their product. We 
may always assume that if n > 1 in ((TJ, then either ipi G A\B and 
ipi+i G B\A, or vise versa. We call this decomposition a canonical 
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form of (p. The group theoretic properties of A *c B imply that if 
n > 1, then (p 7^ 1. From now on we fix the automorphism 

(2) r = (y,x). 

Then the form of the automorphism ip can be replaced by 

(3) ip = p n r--- rpirpo, 

where p , pi, . . . , p n G B and only p and p n are allowed to belong to 
A, see for example p. 350 in 3J. Using the equalities for compositions 
of automorphisms 

(ax+p(y),/3y+i) = (x+a~ 1 (p(x)-p(0)),y)(ax+p(0), (3y+>y), 7 G K, 

(ax + £,(3y + r])r = (f3y + r),ax + £), £,7? G K, 

we can do further simplifications in (J3J), assuming that pi = (x+pi(x),y) 
with Pi(0) = for all z = 1, . . . , n. 

In the next considerations we work in the free algebra K(x,y). We 
denote by \u(x, y) \ the homogeneous component of maximum total de- 
gree of the nonzero element u(x,y) G K(x,y). Following Cohn |3j, p. 
357, we call u(x,y) biased if degj-u| > degjwl. 

Let V = span([x, y] k \ k > 0) be the subspace of K(x,y) spanned 
by all powers of the commutator [x,y]. Since AutK(x,y)(V) = V, the 
group AutK (x, y) acts on the factor vector space K(x, y) = K(x, y) /V. 
Since V is also graded, K(x,y) inherits the grading of K(x,y). Hence 
for the nonzero element u(x,y) G K(x,y) we may define deg u, deg x u, 
degyll, and \u\. Again, u(x,y) is biased if degjw| > degjjwl. 

The following result is a corollary of a lemma of Lane. 

Proposition 1. (Corollary 9.6, pp. 361-362 in 0) Let 7^ u(x,y) G 
K(x,y) and let p = (ax + p(y),/3y + 7) be a nonaffine triangular au- 
tomorphism ofK(x,y). Then each of the following statements implies 
the next: 

(i) u(x, y) is biased; 

(ii) deg u < deg(rpw); 

(iii) deg u < deg(rpu); 

(iv) rpu = u(ay + p(x), /3x + 7) is biased. 

The following consequence of the proposition is the main step of 
the proof of Theorem 6.9.7, p. 361 in [3 j. We include the proof for 
convenience. 
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Corollary 2. Let <p = p n r ■ ■ -Tpirp G AutK (x,y) be written in the 
form J3J). Letu(x,y) G K(x,y)\V and let 

d-i = deg u, d n = deg(pz), 

dj = deg(rpjT- ■ - rp^pou), j = 0, 1, . . . , n - 1. 

If Pi, ■ ■ ■ , Pk o,re all nonaffine automorphisms in the decomposition 

i < 1, k > n — 1, then there exists an integer m between i and k such 

that 

d-\ = rfj-i > di > ■ ■ ■ > d m < d m+ i < • • • < d)~ — d n . 

Proof. Clearly, affine automorphisms preserve the degree in K(x,y). 
If po is affine, then i = 1 and gLi = deg u = deg(poll) = deg(rpoll). 
Similarly we conclude that c4 = d n . Let m > i be the largest integer 
such that di > di + \ > ■ ■ ■ > d m . Hence either m = k or m < k and 
d m < d m+ \. Applying consecutively parts (iii) => (iv) and (i) => 
(ii) of Proposition [T] we obtain that rp m+ iT ■ ■ ■ rpirp u is biased and 
d m+ i < d m+2 - We complete the proof by obvious induction. □ 

We shall also need the following well known lemma, see for example 
Lemma 5.1 in p]- 

Lemma 3. As a vector space K(x,y) has a basis consisting of the 
elements 

(4) u ab = x ai y bl [x,y]x a2 y b * ■ ■ ■ x a ^y^[x, y]x a ^y b ^\ 

where a^, bi, r > 0. 

Note, that the coefficients of u(x,y) G K(x,y) with resepct to the 
basis (HJ) can be found explicitly using the equation yx = xy — [x,y], 
see e.g. the proof of the lemma in [H] for details. 

Corollary 4. Let the element u(x,y) in K(x,y)\V be written as a 
linear combination 

U i X , ii) = ^ labU ab , lab & K, 

of the basis ^ and let p = (ax+p(y), f3y + i) be a nonaffine triangular 
automorphism of K(x, y) . If ai = ■ ■ ■ = a q+ i = for all summands u a b 
with nonzero coefficients •y a f ) , then deg u = deg(pu). // some is not 
equal to and deg p(y) = k > deg u, then deg(/7u) > k. 
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Proof. We use the idea of the proof of Theorem 5.2 in If all ai are 
equal to 0, then x participates in u(x,y) in commutators [x,y] only. 
Since [ax + p(y) , fly] = a(3[x,y], we obtain 

pu = la b a r p r ((3y + 7 ) bl [x, y] {(3y + 7 ) fe2 [x, y] ■ ■ ■ [x, y] {(3y + 7 ) br+1 . 

Hence deg u = deg(pw). Now, let some a* be not equal to 0. Let 

p(y) = 5 y k H h 6 k -iy + S k , 5i G K, 5 ^ 0, and deg p = k> deg u. 

We order the elements u a b from the basis (jlj) lexicographically assuming 
that y > x. The leading monomial of pw a t, = w a b(aa; + p(y),/3y + 7) is 

(-l) r a r p B+r -f kA y kai+bl+1 xy ka2+b2+1 x ■ . . y^+W+i xy a r+1 +b r+1+ i ^ 

A = a i, B = bi- Since k > deg u > A + B + 2r, we obtain that the 
different pu a b have linearly independent leading monomials. If a, > 
for some i, then the leading monomial of pu a b has different degrees 
with respect to x and y. Hence, the corresponding bihomogeneous 
(homogeneous in x and in y) component does not belong to V. Since 
deg(puab) > kA + B + 2r and there exists a nonzero a^, we conclude 
that deg(pw) > k. □ 

Finally, we need some facts from the theory of Grobner bases. 

Proposition 5. Let K be an algebraically closed constructive field and 
let fj(ti, . . . , i?v), j = 0, 1, ... , M, be a finite set of polynomials in 
K[ti, . . . ,tjv]- There is an algorithm which decides whether or not the 
system 

f j (t 1 ,...,t N ) = 0, j = l,...,M, 
has a solution (£1, . . . , £n) e K N such that fo(£i, • • • , £tv) 7^ 0. 

Proof. The Hilbert Nullstellensatz gives that the system 

f j (h,...,t N )=o, j = i,...,m, 

has a solution if and only if the ideal / of K[ti, . . . , tjv] generated 
by fj(ti,...,tx), j = 1,...,M, does not coincide with the whole 
K[ti, . . . , tjv]- We can decide whether or not I = K[ti, ■ ■ ■ , tjv] calculat- 
ing its Grobner basis. If for every solution . . . , ^^r) of the system we 
have fo(£i, ■ ■ ■ , 6v) = 0, then the Hilbert Nullstellensatz again implies 
that some power of /o(ti, • • • , £iv) belongs to I and /o belongs to the 
radical Rad(J) of /. There is an algorithm which uses Grobner bases 
and decides whether or not /o(£i, • • • , £jv) G Rad(J), see for example P 
or the algorithm RADICALMEMTEST, p. 268 in 0. □ 
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3. The main results 

The following two theorems are the main results of this paper. 

Theorem 6. Let K be an algebraically closed constructive field and 
let u(x,y),v(x,y) G K(x,y). Then there is an algorithm which decides 
whether or not v = ipu for some ip = (f(x,y),g(x,y)) G AutK(x,y) . 
The elements f(x,y) and g(x,y) which determine p can be expressed 
in terms of the solutions of systems of algebraic equations. 

Proof. We want to find ip = (/, g) G AutK(x, y) such that v = ipu. We 
can decide efficiently, presenting u and v as linear combinations of the 
basis (JH) in LemmaEl whether or not u(x,y),v(x,y) G V. 
Case 1. If 

in 

u(x,y) = J2 X k[x,y] k G V, X k EK, 

k=0 

then v = ipu is impossible if v G" V. Let 

m 

Since <p[x, y] — u>[x, y], uj G K*, the action of ip on u is determined by 
the linear components f\ = £ix + and g\ = r\\x + r/2y of / and g, 
respectively. Hence 

m 
k=0 

Therefore, we have to decide whether or not the equations 

t k {oj) = X k u k - fi k = 0, k = 0, 1, . . . ,m, 

have a common solution. This can be handled efficiently, determin- 
ing with the Euclidean algorithm the greatest common divisor of the 
polynomials t k (u). It is easy to see that automorphisms ip which 
send u to v can be characterized in their normal form (J3J) as fol- 
lows. For any common solution Uq of the equations t k (uj) = and 
any n>0we define in an arbitrary way pi = (x + Pi(y), y), Pi(0) = 0, 
i — 1, . . . , n. Then we choose a G K*, p{y) G K[y], 7 G K, and define 
p = (ax + p(y), a^LOoy + 7). 

Case 2. Now we assume that u, v G" V. We repeat the main idea 
of the proof of Makar-Limanov, Shpilrain, and Yu ^U] of the result in 
the commutative case. We search for ip in the form ip = p n r ■ ■ ■ Tp\rp§. 
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We can efficiently present u(x, y) and v(x, y) in the form u = u' + uy, 
v = v' + Vy, where uy,vy G V are the sums of the bihomogeneous 
components of u and v which are equal, up to multiplicative constants, 
to powers of the commutator [x, y\. In the notation of Corollary we 
define 

d-i = deg u, d n = deg(v), 

dj = deg(rpiT- ■■Tp 1 rp u), j = 0, 1, . . . , n - 1. 

We assume that both p an d Pn are affine. The other cases are similar 
and also have to be considered. Hence i = 1, k = n — 1. Since d_i 
and d n are equal to the degrees of u and v, they are fixed. Hence there 
is a finite number of choices for the sequence of positive integers dj, 
j = 0, 1, . . . , n, with the property that 

a_i = d > d\ > ■ ■ ■ > d m < d m+ i < ■ ■ ■ < d n _i = d n . 

This also bounds n from above by n < deg u + deg v. We have to 
consider all possible sequences {dj}. We fix one of them. We consider 
the first and the last automorphisms po = (£o£ + £q2/ + £q, VoU + Vo) an( i 
Pn = ((.nX + i'nU + £"5 VnU + ^n) with unknown coefficients £j, r/j, and all 
other automorphisms pj = (£jX + Pj(y),r}jV + t}'a) with unknown r/^ 
and unknown polynomials p^. 

Part 1, Step 1. If, writing y) as linear combination w = ^2 lab^ab 
of the basis (J1J), we have a r = for all 7 a fe 7^ 0, then p u shares the 
same property. Hence rp u is biased and by Proposition [H do < d\. 
Hence m = and we go to the next part of the procedure. We as- 
sume that there exists a nonzero dj. It is easy to see, that the same 
holds for some basis element in the expression of p^u. Then Corollary 
El gives that the degree of px(y) is bounded by the degree of u(x,y). 
Let p\(y) = u>d y d ° + • • • + ujiy + uo, where uq, u>i, . . . , Ud are unknown 
coefficients. This bounds the degree of p\rp§u from above in terms of 
do, e.g. deg(pirpou) < d§. We write pirp u in the form 

pyrppu = ^ 5jZ h z i2 --- z is , z {j = x,y, 

where the coefficients 5i = 5(£,r),u) are polynomials in ^j,T]j,ujj. Now 
we use the equalities deg (pYrpou) = d\ and deg(piTp u) < djj. The 
monomials Zi t z i2 ■ ■ ■ Zi s G K(x,y) of degree s > d^ do not partici- 
pate in pyrpou. If d\ < s < d^ and deg x (z il z i2 ■ ■ ■ z is ) is different from 
deg y (z h z i2 ■■•z is ), then 5 { = 0. If deg x (z il z i2 • • • z is ) = deg x (z h z i2 ■ ■ ■ z is ) 
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for d% < s < c?q, then we write the corresponding bihomogeneous com- 
ponent in the form ^Si'Z^z^ ■ ■ ■ z is = $[x, y] s ^ 2 with unknown coeffi- 
cient $ and again obtain equations of the form 5i = or 5i — ±$. In 
this way we obtain a finite system of algebraic equations 

(5) A ff (£,77,o;,0) = O, q = l,...,Q. 

We want to decide whether or not the system has a solution with 
the property that deg(p\Tp u) = d\, the coefficients £o, Vo> £i, Vi are 
nonzero, and the polynomial pi(y) is of degree > 2. This can be done 
effectively using Proposition 

Step 2. We repeat Step 1 with the element rpiTp u instead of with 
u. If the element rpirp u is biased, we have m = 1 and go to the 
next part of the procedure. If rpirp u is not biased, then we bound 
from above the degree of the polynomial P2{y) in the definition of P2 
and, continuing as above, add new equations to the system of algebraic 
equations 

We continue till the m — l'st step, and obtain the polynomial 

Wi = Pm-lT ■ ■ ■ PiTPqU. 

when d m < d m+ i. We finish this part of the procedure. 

Part 2. We start a similar procedure with v(x,y), applying to it 
pj r ■ ■ ■ TPn^Tp^ 1 for j = n, n — 1, . . . , m if d m < d m+ \ and for j = 
n, n — 1, . . . , m + 1 if d m — d m+ i. 

Part 3. If d m < d m+ i, we obtain the element 

w 2 = pT„]r ■ ■ ■ rp-^rp^v, 

and a system of algebraic equations depending on the unknown coeffi- 
cients of pi. Since w± = rw2, we obtain one more relation between the 
coefficients of po, pi, . . . , p n . If d m = d m+ \, then we consider 

W2 = P~\iT ■ ■ ■ rp-^rp^v. 

Then w 2 = rp m rwi and we have two possibilities. If, writing twi as a 
linear combination twi = l' a b u ab of the basis (HJ), we have cij = for 
all 7^ 0, then 

twi = iabV bl [x, y]--- [x, y]y hr+ \ 

pmTW! = ^ l' ab CmV r m(Vmy + Vmf 1 [x, y\'" [x, y] (VrnV + Vm)^ ■ 

Hence the result does not depend on the polynomial p m (y), we can 
choose it to be arbitrary. The corresponding algebraic system does not 
depend on its coefficients. If some cij is positive, then we bound the 
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degree of p m (y) and determine whether or not the obtained system has 
a solution with nonzero £j, rjj and nonlinear Pj(y). □ 

Theorem 7. Let K be an algebraically closed constructive field and let 
u(x,y) G K(x,y). Then there is an algorithm which decides whether 
or not u is a semiinvariant of some (p G AutK(x,y) , and (p can be 
expressed in terms of the solutions of algebraic systems. 

Proof. Let ^ u(x,y) G K(x,y). We want to find ip G AutK(x,y) 
and a constant A such that p>u = Xu. If u(x,y) G V, then the action 
of cp — (f,g) is determined by $ = £1772 — farji, where f\ = £\x + fey 
and g\ = rjix + r)2y are the linear components of / and g, respectively. 
In particular, u is stabilized by any <p with 1? = 1. We can find all 
possible values of d as in the first part of the proof of Theorem |3 If 
ipu = J^fcLo ^k$ k [x, y] k , then <pu = Xu if and only if A = $ h for all k 
such that Afc 7^ 0. If u(x, y) G" V, then the theorem of Lane implies that 
ip = ifj~ 1 p^p for some triangular or afline p and some ip G AutK(x, y). 
We write ip in the form (J3J), ip = p n r ■ ■ ■ rpirp . If p is triangular, we 
have 

ip = (p n r ■ ■ ■ Tp 1 Tp Q y 1 p(p n T ■ ■ ■ rpirpo) 
= Po 1t Pi 1t ■ ■ ■ r (Pn 1 PPn)r ■ ■ ■ pirp . 
If p is nontriangular afline, then it has the form p = p"rp' some some 
afline triangular p', p" and we proceed in a similar way. Then we com- 
plete the proof as in the second case of the proof of Theorem El □ 

Remark 8. The unitarity of K(x,y) is not essential. The same proofs 
work in the free nonunitary associative algebra in two variables. 

4. Applications to commutative case 

From now on we work in the polynomial algebra K[x,y] over an 
arbitrary field K, keeping some notation from the case K(x,y). If 
7^ u(x,y) G K[x,y], we denote by \u\ the homogeneous component of 
maximum total degree and say that u is biased if degjw| > deg y |it|. 

One of the main steps in the approach of Makar-Limanov, Shpilrain, 
and Yu [10J is Lemma 2 in ^21- A minor modification in its proof 
allows us to simplify the proof of the theorem for the existence of a 
nontrivial stabilizer of u G K[x,y]. 

Proposition 9. Let u(x,y) G K[x,y]\K be biased and let p = (ax + 

p(y), /3y + j) be a nonaffine triangular automorphism. Then deg(pw) > 
deg u. 
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Proof. For simplicity of the exposition we assume that a = (3 = 1 and 
p(x) = x k + nk-ix 1 + • • • + ttiX + 7r is monic, with k > 2. Let the 
homogeneous component of maximum degree of u(x,y) be 

|u| = la x a y b + la ^x a ~ l y b+l + ■■■ + ^xy^ 1 + la+b y a+ \ la ± 0. 

Since u(x,y) is biased, we have a > b. Define a (k, l)-grading on 
K[x, y] assuming that deg^ -^x = k, deg^ kl ^y = 1. Let u m (x,y) be the 
homogeneous component of u of (k, l)-degree m, and let 

u ka+b (x, y) = (3 d x d y 3 + Pd-iX d ~W +k + ■ • • + ^W +k{d ~ e \ Pd, Pe + 0. 

Then, over the algebraic closure K of K , the (/c, l)-homogenity of Uka+b 
implies the decompostion 

t 

u ka+ b(x, y) = £x q y r {x - y k ) s J[(x - X lV k ), 1 ^ K E K. 

i=i 

Clearly, 

deg u ka+b = q + r + k(s + t) = e + (j + k(d - e)) 

> e + 1 + (j + k{d - e - 1)) > • • • > d + j. 

Since k > 2, the only summand of maximum total degree contained 
in Uka+b is r y a x a y h . We conclude that r y a x a y h = f3 e x e y^ +k ^ d ~ e)> and this 
implies that 

(a, b) = (e, j + k(d - e)) = (q,r + k(s + t)), 

a + b = deg u = deg u ka+b = q + r + k(s + t). 

As in the proof of Lemma 2 [TUj, the first step is to show that deg Uk a +b(x+ 
y k ,y) > deg Uka+b(x,y). Let us assume that the oposite inequality 
deg u ka+b (x + y k ,y) < deg u ka+b (x, y) holds. Since 

t 

u ka+b (x + y k , y) = £{x + y k ) q y r x s - (A* - l)y k ), 

i=i 

and k > 2, we derive that 

deg u ka+b (x + y k ,y) = kq + r + s + kt < q+r + k(s + t) = deg u ka+b (x,y), 

{k - l)q < (k — l)s, q<s. 
Since u(x,y) is biased and 

q = a>b = r + k(s + t), q > 0, r, s, t > 0, 

we obtain that q > 2s. This is a contradiction because we already 
have s > q > 0. In this way deg u ka+b (x + y k ,y) > deg u(x,y). Since 
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the leading (k, l)-components of x + p(y) and y + 7 are x + y k and y, 
respectively, we derive that 

u m {x + p(y),y + 7) = u m (x + y k , y) 
+ (k, l)-homogeneous components of lower (k, l)-degree. 
Hence the monomials of Uk a +b(x + y k , y) can vanish in u(x+p(y),y+j) 
only if they cancel with some monomials from u m (x + p(y),y + 7) for 
m > ka + b. Let tuq be the (k, l)-degree of u(x,y). If mo = ka + b, then 
the monomials of Uk a +b{x + y , y) do not cancel with anything. Hence 
deg(pw) > deg Uk a +b(x + y h , y) > deg u(x, y). So, we may assume that 
m > ka+b. Again, the leading (k, l)-component of u mo (x+p(y), y+7) 
is u mo (x + y k ,y). If 

to 

u mo (x,y) = ^ qo y ro {x - y k r H(x - Ky k ), A- ^ 1, 

i=i 

then the leading (k, l)-component of u mo (x + p(y),y + 7) is 

t 

u mo (x + y k ,y) = + y k ) qo y ro x so J]> - (A- - l)y k ) 

1=1 

and does not cancel with other elements of u(x + p{y),y + 7). In 
particular, deg u(x + p(y),y + 7) > deg u mo (x + y k ,y). We have the 
inequalities 

(6) a + b = deg u(x,y) > deg u mo (x,y) = q + r + A;(s + t ), 

(7) kq + r + fe(s + to) = m > ka + b. 
The sum of (JUJ) and (J7J) gives 

a + b + kq + r + k(s +t ) > ka + b + q + r + k(s + to), 
(k - 1)50 > (k- l)a, q > a, 

If we assume that 

(8) a + b = deg u(x, y) > deg u mo (x + y k , y) = kq + r + s + kt , 
then the sum of (JTj) and (jHJ) implies 

a + b + kq + r + fc(s + *o) > ^ a + & + kq + r + s + kt , 
(k — l)so > (k — l)a, s > a, 

and @ gives 

2a > a + b > q + r + k(s + t ) > qo + s > 2a, 
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which is impossible. Hence 

deg u(x+p(y),y + >y) > deg u mo (x + y k ,y) > deg u(x,y). 

□ 

Proposition El implies immediately commutative analogues of Propo- 
sition and Corollary |21 We shall state the first of them. 

Corollary 10. Letu(x,y) G K[x,y]\K and let p = (ax + p(y) , j3y + 7) 
be a nonaffine triangular automorphism of K[x,y}. Then each of the 
following statements implies the next: 

(i) u(x, y) is biased; 

(ii) deg u < deg(rpti); 

(iii) deg u < deg(rpw); 

(iv) rpu = u(ay + p(x), (3x + 7) is biased. 

Proof. The only part of the proof left is the implication (iii) (iv). 
If some v(x,y) G K[x, y] is not biased, then v(y,x) = tv is. Hence, 
if rpu is not biased, then pu is and Proposition El gives that deg u = 
deg(p~ 1 (pu)) > deg(p-u) = deg(rpw) which is a contradiction. □ 

Now we can prove easily the theorem of Lane [E] and Makar-Limanov, 
Shpirain, and Yu [Ej . 

Theorem 11. If K is any field and the automorphism (p of K[x,y] is 
not conjugate to a linear or triangular automorphism, then any semi- 
invariant u(x,y) G K[x,y] of ip is a constant. 

Proof. Let u(x, y) G K[x, y]\K and let G be the subgroup of AutK[x, y] 
which stabilizes the vector space spanned by u(x,y). Writing <p G G 
in the form ©, V 9 = Pn T ' 4 4 t Pi t Pq an d applying the commutative 
analogue of Corollary we obtain that the length n in the expression 
of ip is bounded by 2 ■ deg u. Now the proof is completed by the well 
known theorem in group theory (see e.g. Theorem 6.8.7, p. 351 jSJ), 
which states that if G is a subgroup of A*c B and its elements are of 
the form 

g = a m b m ■ ■ ■ ai&i, a f G A, G B, 

where the integers m = m(g) are bounded by the same n for all g G G, 
then G is conjugate to a subgroup of A or B. In order to replace the 
affine automorphisms with linear ones, we need to use the fact that 
Aut K[x,y] is also a free product of the linear group GL 2 (K) and the 
triangular group B with amalgamation over their intersection. □ 
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Clearly, we have analogues for K[x,y] of the algorithms described 
in Theorems |H1 and [7| (compare the first algorithm with this of Makar- 
Limanov, Shpilrain, and Yu PU|). in particular, when K is an al- 
gebraically closed constructive field, we can decide whether or not 
u G K[x, y] is a semiinvariant of some (p G AutK[x, y] and to express 
<p in terms of solutions of algebraic systems. 

Remark 12. Clearly, over an algebraically closed field K any linear 
automorphism can be triangularized. Smith [13] has determined the 
eigenvalues and the eigenvectors of any triangular automorphism p of 
K[x, y] when charK = 0. Up to conjugation, the possibilities are: 

(i) p = (ax,f3y), u(x,y) is a linear combination of monomials x n y m 
with the same value of a n f3 m ; 

(ii) p = (ax, (3y + 7), 7 7^ 0, u(x, y) does not depend on y and is a 
linear combination of powers x n with the same value of a n \ 

(iii) p = (ax + p(y) , fly) , p(y) 7^ 0, u(x, y) does not depend on x and 
is a linear combination of powers y m with the same value of (5 m . 

If u = w(f) for some coordinate f(x,y) and some polynomial w(z), 
we have 

du du 

and the ideal of K[x, y] generated by f x and f y conicides with the whole 
K[x,y]. Hence the greatest common divisor of u x and u y is w'(f) and 
this can be used to determine whether or not u is a semiinvariant of 
a nontrivial automorphism in the cases (ii) and (iii). We cannot see 
how to handle directly the case (i), i.e., to determine whether or not 
u = w(f,g) with some specific properties of the polynomial w(z,t). 
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